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MEDIA ARTICLE COMPOSITION 



The present invention relates to a method of, and apparatus for, composing a media 
article. 



Media articles portray content (whether real, imagined or computer-generated) to a 
person's senses. Media articles can be presented to a person via a variety of media, 
including text, voice, sound, pictures or moving images. 

10 As recording technologies have improved, the amount of recorded media articles 
available to a consumer has grown rapidly. Media articles are often recorded in 
media files ( note that although the plural 'media' is used here, 'media file' is to be 
understood to include both files which are intended to be conveyed to user by only 
one medium - e.g. text or speech and also 'multimedia' files whose meaning is 

15 conveyed by a plurality of media). The Internet is the most recent communications 
network to emerge and provides worldwide transmission of recorded digital media 
files representing text, sound, pictures, moving images or a combination to these. 
Since the number of media files accessible via the Internet is so large, there is a need 
to label media files with some description of what they contain. Thus, for example, 

20 HTML files contain 'meta' tags which include keywords which indicate what subjects 
are covered in the web-page presented to the user. 

A proposal for adding metadata to video files (the 'Multimedia Content Description 
Interface' more widely known as MPEG-7) is being discussed by the Moving Pictures 
25 Expert Group. 

One method of composing a media article involves the putting together of a plurality 
of components. For example, a film is made up of a plurality of scenes as set out in 
a screenplay. There have been some attempts to create a media article automatically 
30 in this way. For example, the production of a Waltz in accordance with a musical 
dice game is found at http://sunsite.univie.acat/M02art/dice/ . Similarly, an 
automatic story generation program, the Romance Writer is available from at 
http://familyqames.com/features/humor/romance.html . 
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According to a first aspect of the present invention, there is provided a method of 

composing a media article comprising: 

analysing digital metadata associated with a first media file, which digital 
5 metadata includes: 

related file identity data identifying a second media file; and 

relationship data which indicates the relationship between what is 

represented by the first media file and what is represented by the second media file; 

and 

10 arranging said first and second media files in a media article in accordance 

with said analysis. 

By analysing digital data associated with a first media file, which digital data includes 
an identifier of a second media file and an indication of the relationship between what 
15 is represented by the first file and what is represented by the second file, and 
arranging the first and second media files in a media article in accordance with said 
analysis, a method of composing a media article is provided which obviates the need 
for another source of sequencing information to be provided at the time the media 
article is composed. 

20 

Digital media files are stored in a variety of formats. The expression media file is not 
intended to be limited to any particular format and so includes, for example, a file 
which merely contains data on the position of components seen by a user when 
playing a computer game - the data in that file subsequently being processed by 
25 rendering software to generate an image for display to the user. 

In some embodiments, said method further comprises generating said file identity 
data and said relationship data. 

30 Preferably, said metadata further comprises data characterising the content of said 
media files, said method further comprises the step of selecting a subset of media 
files including said first and second media files from a set of media files. 




3 

This combination of a) searching to provide a plurality of potential components for a 
media article, or part of a media article; and b) subsequently arranging a plurality of 
the selected components in accordance with metadata associated with those 
components allows the automatic composition of media articles directed towards a 
5 particular theme more easily than has hitherto been possible. 

Preferably, the method further comprises making a plurality of such selections; and 
concatenating the results of said selections. 

10 This allows an approach to the composition of a media article which broadly follows 
an established pattern, but which allows a degree of flexibility within that pattern not 
seen in conventional systems. Thus the type of broad pattern known for narratives 
and films - for example, the StoryCraft program from StoryCraft Corporation, 560 
Roland Drive, Norfolk, VA 23509, USA helps an author write a story by asking the 

1 5 writer to introduce the hero and an antagonist, prior to some conflict between them, 
which is then followed by the hero's triumphal return home - can still be used, but 
variations within that pattern can easily be introduced by changing the nature of the 
selection. 

20 This will allow, for example, the cost effective creation of different versions of a film 
or computer game, conforming to different artistic or qualitative ambitions, whilst still 
retaining subjectively high standards of narrative, motion and audio continuity. 

According to a second aspect of the present invention, there is provided a media 
25 article composition apparatus comprising: 

one or more memory devices storing, for each of a plurality of media files, metadata 
including relationship data indicating one or more relationships between the media 
represented in said media file and the content represented in one or more other media 
30 files; and 
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one or more processors in communication with said one or more memory devices and 
arranged in operation to compose a media article by arranging said media files or 
identifiers thereof in accordance with said relationship data. 

5 In preferred embodiments, an object-oriented database is used to store objects 
containing metadata associated with a media file identified by said metadata. 
Relationship metadata can then be represented by relationships between objects in 
the object-oriented database. 

10 By way of example only, specific embodiments of the present invention will now be 
described with reference to the accompanying Figures in which: 

Figure 1 is a schematic illustration of components of a personal computer which may 
be operated to provide a media assembly system in accordance with a first 
15 embodiment of the present invention; 

Figure 2 is a diagram of the architecture of a computer program used in controlling 
the personal computer of Figure 1 in accordance with a first embodiment of the 
present invention; 

20 

Figure 3 shows metadata associated with a media element following the use of the 
media mark-up tool of Figure 2; 

Figure 3B shows a cause object stored in the object-oriented database of Figure 2; 

25 

Figures 4A to 4G show how an editor builds the relationship data included in the 
metadata of Figure 3; 

Figure 5 is a hierarchical representation of the relationship data entered by the editor 
30 as shown in Figures 4A to 4H; 

Figure 6 shows template data generated using the template creation tool of Figure 2; 





Figure 7 shows user profile data generated using the user profile creation tool of 
Figure 2; 

Figure 8 is a flow chart showing the operation of the template populator component 
5 of the first embodiment of the present invention; 

Figure 9 shows the operation of the template populator in relation to the first section 
of the template illustrated in Figure 7; 

10 Figures 10A to 10D show the operation of the template popu.ator in relation to the 
second section of the template illustrated in Figure 7; and 

Figure 1 1 shows an edit decision list as might be produced by the template populator 
module. 

15 

Figure 1 shows a personal computer which comprises well-known hardware 
components connected together in a conventional manner. The well-known 
hardware components comprise a central processing unit 10, random access memory 
12, read-only memory 14, a hard disk 16 and input/output devices 18,20,22,24, and 
20 26. The hardware components are interconnected via one or more data and' address 
buses 28. The input/output devices comprise a monitor 18, a keyboard 20, a mouse 
22, a CD ROM drive 24 and a network card 26. The network card is connected to a 
server computer 30 by the public Internet 32. 

25 Figure 2 shows the software and data components of a system according to a first 
embodiment of the present invention. The software includes three input program 
modules, namely a media mark-up too. program 40, a template creation tool program 
42, a user profile creation tool program 44, object-oriented database management 
software 45, and two output program modules, namely a template populator program 

30 module 46 and a content synthesis program module 48. 

The system also includes two stores for the persistent storage of data. The first of 
these is a content store 50 which comprises a number of media files stored on the 
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hard disk 16 using the file system provided by the computer's operating system 
program. The second store for the persistent storage of data comprises an object- 
oriented database 54 known as ObjectStore® supplied by Excelon Corporation, 
Burlington, Massachusetts. The database stores three different categories of objects, 
5 namely media objects 51 (each of which includes metadata describing one of the 
media files stored in the content store 50), template objects 52 and user profile 
objects 53. The objects in the database are again stored on the hard disk 16 of the 
computer. Persistent storage for the object-oriented database and/or the content 
store might, instead of the hard disk, be provided by removable data carriers such as 
10 DVDs, CDs, CD-ROMs or on different computers (accessed for example via a URI) 
accessible via a network connection such as is provided by the network card 26. 

The three input program modules control the computer (Figure 1) to provide a user 
interface allowing the generation of objects which include data structures 

15 constructed in accordance with predetermined structured data models. This 
increases the ease with which different program modules can exchange data. Where 
the structured data model is made public, different parties can write different 
components of the program which are nevertheless able to exchange data with one 
another. Also, objects created by one editor can be used by another editor working 

20 at a different location or at a later time. 

In the present embodiment, the three input program modules offer the following 
functionality: 

25 The media mark-up tool provides an interface for a editor to update the content store 
50 and the object-oriented database 54. In practice it is envisaged that an editor 
using the present embodiment will have access to media elements generated by other 
editors, rushes from various sources, sections of prepared programmes, still 
photographs and various other pieces of media (all represented in electronic form) at 

30 his disposal. These media elements are stored in an appropriate directory structure in 
the content store. Each directory is known as a 'bin 1 in the art - a reference to the 
labelled bins in which rolls of film relating to a particular project are stored. 
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Media elements are often recorded digitally in a file which may contain a number of 
elements in a linear stream. Such a file may therefore contain one or more media 
elements associated with it. The media mark-up tool allows the editor to preview 
files and, where there are to be several media elements within that file, set start and 
5 end points defining the scope of each media element within the file. If there is only 
one media element associated with the file then the start point is simply zero and the 
end point is the length (duration) of the media. In this way, a editor is able to 
generate a plurality of files, each forming one media element. The editor gives a 
name to each file at the time he defines the start and end points of the media 
10 element. 

However, for the purposes of the present description, it is assumed that the editor 
begins only with a file that includes an electronic representation of unedited film 
recorded at a football match and introduction sequences for a football programme 
15 etc. An unedited piece of film is known as a 'rush 1 in the art. Using the media mark- 
up tool 40, the editor might select various sections of the rush and store each as a 
media element in a shorter f ile in a directory in the content store 50. 

The media mark-up tool also provides a tool enabling the editor to generate or edit 
20 metadata for media elements stored in the content store 50. The tool stores this 
metadata as an object in the object-oriented database 54. 

On selecting a directory within the content store, the editor is provided with a 
graphical user interface which presents set of icons (Figure 4A), each of which 

25 represents one of the media elements held in that directory. These icons may just be 
a still from a video sequence contained within the file, for example. In the examples 
below, they are illustrated as boxes which include the media element identifier 
entered by the editor in the top right-hand corner and which contain a description of 
what occurs in the media element in the centre of the box. The user interface allows 

30 the editor to select the media elements using the mouse 22 and to move the media 
elements around the screen using the mouse. 
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Having selected one of the media elements, the editor enters metadata to be 
associated with that media element in two stages. In a first stage, the editor can 
double-click on one of the pictures to bring up a form onto which the values of the 
parameters included within the schema can be entered. 

5 

An example of the metadata generated in the first stage is shown in the second to 
twelfth row of Figure 3 (the information in the first row having been generated when 
the editor gave a media element identifier to the file). 

10 It will be realised that the metadata is arranged in accordance with a structured data 
model. In each row, the entry at the rightmost column represents the value of a 
property which is input by the user. The structured data model may provide that a 
plurality of properties should be labelled as members of a unit at a first level of 
aggregation - here referred to as a set of properties (column second from the left in 

15 those rows which have four columns). The structured data model may also provide 
that a plurality of sets should be labelled as members of a unit at a second level of 
aggregation - here referred to as a superset of properties (leftmost column in those 
rows which have three or four columns). Those skilled in the art will realise that 
further levels of aggregation might be provided. 

20 

The hierarchical arrangement is influenced by the Multimedia Content Description 
Interface mentioned above. The intention is not to enforce usage of a complete data 
model across all possible applications, but to enable re-use of content within the 
subject domain of a production company or a specific set of projects (eg. wildlife 
25 documentaries). The data model provided is intended to provide a maximal set of 
elements and an interface which assists their use and the vocabularies which can be 
applied to them. 

The metadata includes a variable number of parameters (but must nevertheless 
30 conform with the predetermined structured data model). In the example, shown in 
Figure 3, the editor has entered values for 18 properties. These include: 
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i) Media Element ID - this identifies the media element - in the present example, the 
editor has given it a numerical value of O.xx, where xx reflects the position of the 
media element within the original rush; 

5 This is followed by a 'Media' superset which comprises two properties and a 
'Position' set of properties. The two properties are: 

ii) URI - the Universal Resource Identifier of the file which contains the media 
element; 

10 

iii) Format - this gives an indication of the format of the data making up the file; 

The 'Position' set contains two properties as follows: 

1 5 iv) In - an indication of the time elapsed since the start of the rush at the start of the 
media element; 

v) Out - an indication of the time elapsed since the start of the rush at the start of 
the media element; 

20 

The 'Media' superset is followed by a superset of four 'structural 1 properties. That 
superset begins with 

vi) Description - a description of the content of the file; 



which are followed by another set (called 'Event') which contains three properties: 
vii) Nature - the type of event that is seen in the video sequence recorded in this file; 
30 viii) Performer - the person performing the principal action of the event; 
ix) Recipient - the person subject to the principal action of the event; 
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These properties are followed by a domain-specific superset of properties which are 
only sensibly applied to media elements which relate to material obtained from two- 
sided sporting events; 

5 The first two properties belong to a set (called 'Teams') of two properties: 

x) Home Team - the name of the team playing on their home ground during the 
football match featured in the original rush; 

10 xi) Away Team - the name of the other football team in the football match featured 
in the original rush; 

This set is followed by the two properties: 
1 5 xii) Performer Allegiance - the side (if any) to which the performer owes allegiance; 

xiii) Recipient Allegiance - the side (if any) to which the recipient owes allegiance; 

These two properties are followed by a set (named 'conceptual') containing two 
20 properties: 

xiv) Interest Value - this value, between 0 and 1 indicates how significant the editor 
considers this media element to be; and 

25 xv) Rating - this value indicates the suitability of the media element for showing to 
people based on an age criterion - in a similar way to the classification given to films. 

Once the editor has entered this data, the picture is replaced with the description of 
the media element given as the value of the 'description' property above. The form 
30 is then removed from the display to return to the display of a plurality of pictures 
representing the media elements selected by the editor from the original rush (Figure 
4A). 
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The media element metadata is then stored as a media object in the object-oriented 
database 54. 

The second stage of the metadata creation which generates one or more 
5 'Relationship 1 properties will now be described in relation to Figures 4A to 4H. The 
user is provided with a graphical user interface, allowing him to indicate relationship 
properties between media elements by moving and clicking on icons representing 
those media elements on the screen 18 of the PC (Figure 1), 

10 One relationship which the editor may indicate is a causal relationship. To do this, 
the editor clicks on a button element presented on the screen (not shown), which 
changes the form of the cursor. Thereafter the user moves the cursor to an media 
element which he judges to be a cause of another media element. He then clicks on 
the media element to identify it as a causal media element and moves the cursor to 

15 the media element which represents the effect of that cause and clicks again. Once 
he has done this, an arrow is drawn from the first media element to a diamond shape 
representing a cause object and then a second arrow is drawn from the diamond to a 
second media element. An editor may wish to make a causal association of this type 
when he considers that a viewer seeing the media element representing the effect 

20 would also wish to see the cause. In the example shown in Figure 4B, the editor has 
indicated that: 

i) media element 0.13 is caused by media element 0.12, and that both media element 
0.14 and 0.15 are caused by media element 0.13; and 

25 

ii) media element 0.53 is caused by media element 0.52 which is in turn caused by 
media element 0.51 . 

In response to the input of a causal association, a cause object (Figure 3B) having its 
30 own media element identifier is added to the object-oriented database 54. The 
metadata associated with the media element representing the effect is updated to 
include a parameter indicating the media element identifier of the cause object. In 
more detail, the media object has a linked list of pointers to any cause object which 
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causes it or is effected by it. Cause objects also hold two lists of pointers back to 
the media objects with which they are associated. 

An example is seen in the last parameter shown in Figure 3. The cause object also 
5 includes metadata indicating the media element identifier which causes the effect. 

Several different media objects might cause the same effect (e.g. a hero could die 
because he was poisoned, or because he was crushed to death) and there can be 
several different effects from a cause (e.g. because of the hero's death, the evil 
10 queen lived undefeated until the ripe old age of 103, and the broken-hearted princess 
made a vow of celibacy and became a nun). It is for this reason that a cause-effect 
relationship is represented using a cause object. 

A second type of relationship that the editor may indicate is that of sequence. To 
15 indicate such a relationship, the editor arranges the media elements he wishes to 
group into the same rectangular area of the screen, ordering them from left-to-right in 
accordance with the sequence they should .follow, clicks on a further button (not 
shown), which causes the cursor to change form and moves the cursor to a position 
to one corner of that rectangular area. Thereafter, the editor clicks the button on the 
20 mouse 22, and holds that button down whilst moving to an opposite corner of that 
rectangular area whereupon he releases the button. This results in a thick, solid 
rectangular line being drawn around the media elements contained within the 
rectangular area with a thick, short arrow being drawn in the top left-hand corner of 
the area defined by the rectangular line. In the example in Figure 4C, the editor has 
25 indicated that the media element representing Team B's fifteenth pass is followed by 
the media element representing Team B's sixteenth pass which is in turn followed by 
the media element representing Team B's first goal. 

An editor might wish to indicate a sequential relationship of this nature where he 
30 feels that the media elements should be shown in the indicated order (if more than 
one of the media elements are selected by the template populator module). Media 
elements showing gardening at different times of year, for example, might be 
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arranged into a sequence so that an element representing the garden in spring 
precedes an element representing the garden in summer and so on. 

On creation of a sequence in this way, a sequence object is created in the object- 
5 oriented database as a container object containing the media objects associated with 
the media elements included within the sequence. As will be seen below, it is 
possible to generate a sequence which itself includes sequences. This hierarchical 
property is reflected in the first number in the identifier attributed to the sequence. 
Where the sequence includes only individual media elements, then the sequence 
10 identifier is of the form 1.x where x is simply incremented at each time a new 
sequence or group (explained below) at the first level of the hierarchy is formed. 
Hence the sequence shown in Figure 4C is given the identifier 1.1. 

The media object (i.e. metadata) associated with each media element in the sequence 
15 has the position of the media element within that sequence added to it. The object- 
oriented database also records the fact that each media object is a child of (i.e. is 
included within) the newly created sequence object. An example of the sequence 
position metadata can be seen in the penultimate row of Figure 3. 

20 In this embodiment, there is no metadata giving a description of groups or 
sequences, but there could be - such metadata might, for example, be entered by 
right clicking, selecting properties and entering text in the description field of the 
resulting dialog. 

25 The third type of relationship an editor may wish to indicate between media elements 
is that of membership of a group. An editor might do this where he wishes to 
indicate that in a plurality of the media elements in the group are selected, then they 
should be shown together. A group is formed in the same way as a sequence, save 
for the order of the media elements on the screen being inconsequential and the 

30 editor clicking on a third button (not shown) representing a group prior to defining the 
rectangular area which contains the media elements to be included in the group. 
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This action creates a group object, a container object which contains the media 
objects associated with the media elements within the group. Group objects are also 
stored within the object-oriented database 54. 

5 Figure 4D shows the display seen by a editor after he has formed media elements 
0.13, 0.14 and 0.15 into a first-level group 1.2, 0.39, 0.40 and 0.41 into a second 
first-level sequence 1.3, media elements 0.52 and 0.53 into a second first-level 
group 1 .4. 

10 Figure 4E shows the editor joining the media element 0.51 and the third first-level 
sequence 1.4 into a second-level group 2.1. This creates another group object in the 
database, which is a container object for another group object. In this way a 
hierarchy of container objects can be built up in the object-oriented database 54. 

15 Figure 4F shows the editor placing the two first-level sequences 1.1 and 1.3, the 
first-level group 1 .2 and the second-level group 2.1 into a third-level group 3.1 . 

Figure 4G shows the editor defining a fourth-level sequence. 

20 Figure 5 shows the relationships entered by the editor in hierarchical form and 
therefore reflects the object hierarchy which the editor has built up within the object- 
oriented database 54 on the hard disk 16. 

Returning to Figure 2, the template creation tool 42 provides an interface for an 
25 editor and/or producer to specify the desired characteristics of a media article - thus 
creating a template object (Figure 6). The tool stores the metadata which constitutes 
this template object within the object-oriented database 54. 

Like a media object, a template object for use in the present embodiment conforms to 
30 a comprehensive predefined data model. As can be seen from Figure 6, this 
predefined data model includes a title field, and a plurality of sections. Each section 
is a set comprising a name field, a query field, and, optionally, a constraint field. The 
template populator tool prompts the user to enter a name for the template and to 
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indicate the section structure (top-level sections may themselves contain sections). 
The editor indicates the section structure using a graphical user interface component 
similar to the Folder List provided in Microsoft Windows Explorer for example. In the 
example given in Figure 6, the template has a flat structure of three sections. 

5 

Because the template encodes the media article characteristics using complex queries 
and a potentially deep structure, the editor interface divides the task of template 
creation between a plurality of roles. A person who is assigned an editor role defines 
the top-level structure of the template and a person who is assigned a producer role, 
10 (a producer normally having closer control of the actual product being created), 
refines the structure and adds queries (requests for information from the object- 
oriented database). In particular, as will be explained below, the producer specifies 
the linkages to the user profile thereby defining the 'balance of power' between 
themselves and the consumer. 

15 

The template creation tool provides an object browser which can be used to search 
for existing media objects and template objects. Existing templates can be modified 
and portions of a template can be copied into new templates. 

20 Having defined the section structure using the graphical user interface mentioned 
above, perhaps using the media object browser, the editor / producer is provided with 
a graphical user interface which facilitates the process of query formation. 

The editor uses this graphical user interface to enter query strings for each of the 
25 sections. The query string for the first section in Figure 6 indicates that candidate 
media objects to fill this slot in the template must have the string "Football Intro" in 
their URI. 

The query string for a section can be considerably more complex, as is seen in the 
30 'Main' section of the template of Figure 6. Here the query is written in the XPath 
language - the way in which that query is evaluated will be explained after the user 
profile metadata has been described below. 
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The editor also enters constraints for those sections where he wishes to place some 
constraint on the media elements represented by the media objects retrieved from the 
database in response to the query. Constraints are intended to restrict the way in 
which media objects are assembled by the template populator. Possible examples of 
5 constraints include time (e.g. this section must be 5 minutes long), space (e.g. this 
presentation must be viewed on a 640 * 480 pixel display), or number (there must 
be five news items in the 'headlines' section of a news programme). 

The user profile creation tool 44 provides a user interface for generating user profiles 
10 like the user profile seen in Figure 7. 

The user profile expresses the preferences of a particular user. As with media 
objects and the template objects, the data must adhere to a predetermined data 
structure or schema. The schema dictates that each user is identified by ah identifier 
15 (the numeral 'Y in this case). The 'Structural' element of the user profile indicates 
the things the user likes - in this case, the football team Team B, especially Paulo Di 
Canio and Trevor Sinclair, the Essex cricket team, especially Nasser Hussain, the 
actress Milla Jovovich and the actor Jimmy Nail. 

20 The template populator program module (Figure 8) provides a process for the 
automatic assembly of an edit decision list in preparation for the synthesis of a set of 
media objects into a personalised media article for a consumer. On starting (step 
60), the template populator takes as its inputs a specific template, a specific user 
profile, and an indicator of a store of media objects (in the present case, an indication 

25 of the location of the object-oriented database 54). 

Once the specific template, user profile and store of media objects have been 
specified, the template populator examines (step 61) the template (Figure 6) for any 
queries written in the XPath language. If such a query is found, it is resolved using 
30 the Apache Project's Xalan processor to evaluate XPaths. 



For example, the query reading: 
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( Team is " Iprofile (Profile/Sports//Team) ") AND 
(Action is "Goal") ) 



is resolved, when the user profile in Figure 7 (specified on starting the template 
5 populator program module) is considered, to: 

(( Team is "Team B" OR "Essex" ) AND (Action is "Goal")) 

The template populator then identifies the first section of the template (Figure 8) and 
1 0 iterates (steps 62 to 75) through the template's hierarchical structure. 

Each iteration (steps 62 to 75) involves the next section in the template being found, 
any query in that section being executed (step 62) on the object-oriented database 
54 to return a selection of relevant media objects. 

15 

The first iteration relates to the section named 'Intro' in Figure. 6. The iteration 
begins (step 62) with the carrying out of the query contained with the section (i.e. 
URI contains "Football Intro"). Figure 9A illustrates the selection that results in this 
example. 

20 

Then, in step 64, a tree is constructed which includes the selected media objects as 
its 'leaves'. This construction takes place as follows: The parent object of the first 
selected media object is retrieved followed by its parent object and so forth until an 
object is reached which has no parent object associated with it (the 'Introduction' 

25 object in this example has no parent object, so is the only object included in the tree). 
At this point, a single linked list from the leaf object to the top-level container has 
been reconstructed. Another selected leaf object is examined (if more than one 
object is selected as a result of the query), and the ancestry of that leaf object is 
followed until either an object is retrieved that already exists in the linked list 

30 representing the ancestry of the first object or another top-level container is 
encountered. Repeating this process for all the other objects in the selection 
reconstructs the minimal tree containing those objects. 
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As indicated above, in the first iteration, the resultant tree contains only the 
'Introduction' media object 0.1 . 

The subsequent steps, in the loop of instructions (steps 66 to 72) in the template 
5 populator program which alter the tree data structure which is stored and used in 
generating the edit decision list, have no effect in relation to the first section of the 
template, so will be described below in relation to the second iteration of the loop of 
instructions carried out on the second section of the template. 

10 Throughout the iteration, the tree structure is stored in the PC's volatile memory 1 2. 

At the end of each iteration of the group of instructions, a determination is made 
(step 74) as to whether the final section in the template has been considered. If not, 
the next section is identified (step 75) and next iteration carried out. 

15 

The second iteration is carried out in relation to the central section of the Football 
Goals Highlights template (Figure 6). In this case, the section query is carried out 
using the user profile (Figure 7) in order to resolve the query as described above to 
take account of the user's preferences. 

20 

The query (step 62) results in the selection of the media elements 0.12 and 0.53 
(Figure 10A). A new tree data structure is created as described above and stores 
data representing the tree illustrated in Figure 10B. 

25 Thereafter, the selection of media objects is expanded to take account of cause / 
effects relationships specified by the user (step 66). In detail, this step involves the 
examination of the metadata of each selected media object to find how many cause 
objects are associated with that media object. If no cause objects are found then 
the media object is moved to a list of resolved media objects. If only one cause 

30 object is found, then the cause object is moved to a list of cause objects, and the 
media object is moved to the list of resolved media objects. If more than one cause 
object is found, then each possible cause object is added to a list of possible cause 
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objects (if it is not already present in that list) and the media object is added to a list 
of unresolved media objects. 

In the present example, only one cause object is found (that illustrated in Figure 3B), 
5 so the media object 0.53 is moved to a list of resolved media objects (and added to 
the tree data structure associated with this iteration of the loop of instructions - as 
shown in Figure 10C) and the cause object is moved to a list of cause objects. 

Where a list of possible cause objects is created, the cause object which causes the 
10 most unresolved media objects is found. This cause object is moved into the list of 
cause objects, and the media objects it causes are added to the resolved media 
object list mentioned above. This process is repeated until all the list of unresolved 
media objects is empty. In the present example, this step is not applied. 

15 Each cause object in the list of cause objects is then examined to find how many 
media objects it was caused by. If only one media object causes it, then the cause 
object is moved to a list of resolved cause objects and that media object is added to 
the list of unresolved media objects. If more than one media object causes the cause 
object then those media objects are added to a list of possible media objects if they 

20 are not already present. 

Since, in the present example, the only cause object (COD in the list of cause objects 
is caused by only one media object (0.52), then the cause object is moved to a list of 
resolved cause objects and the media object (0.52) is moved to the list of unresolved 
25 media objects. 

Where a list of possible media objects is obtained, the media object which causes the 
most cause objects is found. The cause object is then moved into a list of resolved 
cause objects and the media object is added to the list of unresolved media objects. 
30 This is repeated until the list of cause objects is empty. 

The above procedure is then repeated for any unresolved media objects (so that 
chains of causation are traced back to the original cause). In the present case 
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therefore the above procedure is repeated for media element 0.52 and results in the 
addition of media object 0.51 to the tree associated with this iteration (Figure 10D). 

The building of the tree (steps 62 to 66) is followed by sorting (steps 68 and 70) of 
5 the objects within the tree. 

The first stage of sorting (step 68) takes account of the sequence information 
entered by the user. The is done by using the known 'Quicksort' algorithm to place 
the nodes of the tree in the correct order as identified by the sequence position 
10 metadata associated with the object. This is done starting at the top of the tree and 
then moving towards the leaves (i.e. the media objects) of the tree. 

The second stage of sorting takes account of cause / effect linkages between the 
members of a group or the descendants (i.e. objects further down the tree) of the 
15 members of a group. Where groups do not have such cause / effect linkages then 
this stage of sorting need not be carried out on those groups. 

The second stage of sorting begins by labelling each member of a group in the tree 
with all the causes and effects attached to it (if it is a media object) or to any of its 
20 descendants (if it is a container). 

Further labels are then added to the object metadata to reflect the logical relation that 
if a causes b and b causes c, then a causes c. The same is done to reflect the logical 
relation that if f is caused by e, and e is caused by d, then f is caused by d. 

25 

The known Quicksort algorithm is then used to ensure that causes are shown before 
effects. As those skilled in the art will know, implementations of Quicksort allow the 
user to define a function which gives the order of two objects passed to it. In the 
present case, a function is provided which indicates that a goes before b is a causes 
30 b and that d comes after c # if d is caused by c. 
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Thus, at the end of the sorting steps (steps 68 and 70) in the second iteration, media 
elements 0.12, 0.51, 0.52 and 0.53 form the leaf nodes of the tree associated with 
the central section defined in the Football Goal Highlights template (Figure 6). 

5 The template populator then evaluates any constraints and updates the tree 
accordingly (step 72). To evaluate a time constraint, the duration of each media 
object included within the tree is calculated by subtracting the 'Out' property from 
the 'In' property, and these durations are added together to reach an actual duration. 
If this duration is found to be greater than the target duration, then media objects are 
10 removed from the tree. If this duration is less than the target duration, then media 
objects are added to the tree. 

in the present embodiment this pruning or growing of the tree is done in accordance 
with the Interest Value metadata associated with the media objects. 

15 

Where the actual duration is longer than the target duration, the following process is 
carried out: 

1 ) If the difference between the target duration and the actual duration is less than 
20 the duration of the shortest media element in the tree, then the process terminates; 

2) Otherwise, a list of the media objects within the tree is created and this list is 
sorted in order of the 'Interest Value 1 property. 

25 3) The media object with the lowest value is removed (assuming that this object is 
not the cause of one or more of the other media objects in the list) from the tree and 
the actual duration re-calculated. The difference between the target duration and the 
actual duration is re-calculated and the above steps repeated until the difference is 
less than the duration of the shortest media element remaining within the tree. 

30 

Where the actual duration is less than the target duration, the following process is 
carried out: 
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A) The query in the section is amended by removing its last 'AND' operator and the 
condition which follows it, or if there is no 'AND 1 operator, by adding an 'OR' 
operator followed by a condition such as 'Interest Value' > 0.6. A new tree for the 
current section is then created in the same way as the original tree. This process is 
5 repeated until the actual duration is greater than the target duration. Thereafter, 
steps 1 ) to 3) above are carried out. 

Once this pruning or growth has been carried out, the second iteration ends. 

10 It will be clear that the third iteration will merely generate a tree comprising media 
object 0.99. 

When all sections have been populated with media object metadata and sequenced in 
accordance with the queries, constraints and user preferences provided, the template 
1 5 populator outputs (step 78) the edit decision list (Figure 1 1 ) by concatenating the 
media elements found at the leaves of the trees generated in the three iterations of 
the loop. 

The edit decision list (Figure 1 1 ) produced by the template populator program module 
20 (46) is passed to the content synthesiser module (48). In the present example, the 
content synthesiser module outputs one scene after another in a streamed video 
presentation or concatenates the scenes together in order to produce a programme 
file. 

25 The content synthesiser provides a process to automatically synthesise a set of 
media elements into a personalised media article for a consumer. The synthesiser is 
similar to a conventional non-linear editing system in that it uses the edit decision list 
(Figure 11) to locate and assemble a set of media elements into a completed media 
article. However, while a conventional editing tool is used only during post- 
30 production and resides on the editor's premises, the content synthesiser is a portable, 
lightweight application which can be deployed at the edge of the network, or actually 
within the consumer's device (such as a PC or personal digital recorder). This means 
that media articles can be synthesised at the point of delivery, with considerable 
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benefits in terms of the level of personalisation which can be achieved. Furthermore, 
changes in any metadata including template, media object or user profile can be 
reflected dynamically in the resulting media article. 

5 When invoked by the consumer, the content synthesiser causes a user profile and 
template to be passed to the template populator, which processes these as described 
above returning an edit decision list to the synthesiser. The edit decision list is then 
parsed to discover which media elements are required and how they interact with 
each other. A timeline comprising various references to various media elements 
10 within the content store 50 is then assembled according to this information. 
Transition effects (examples of which are defined in the ANSI/SMPTE 258M/1993 
standard) are applied to the media where required. Finally, any caption text and 
graphical overlays are parsed and rendered to a static image, which is then added to 
the timeline with a high priority ensuring they are visible above any other graphical 

15 elements. 

Once the timeline has been assembled, the personalised presentation is rendered in 
real time using a suitable media compositor technology, such as Apple Quicktime or 
Microsoft DirectShow. The modular nature of the compositor means that a content 
20 synthesiser could be more easily implemented within an embedded architecture such 
as a set-top box. 

The present invention may be embodied in many different ways. For example, the 
embodiment described above may be altered in one or more of the ways listed below 
25 to provide an alternative embodiment of the present invention (this list is by no 
means exhaustive): 

I) in the above embodiment the input program modules generated data structures 
having a predetermined structure. In some embodiments, the predetermined structure 
30 is provided by a document conforming to the XML Schema guidelines; 



ii) the constraints section in the template might be variable by the user - for example, 
a user could be provided with a graphical user interface in which he can select the 
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duration of the media article he wishes to see. A corresponding value can then be 
added to the template object by the template populator program; 

iii) in the above-described embodiment, one or more causal media objects causing a 
5 media object included in the selection were added to the tree. In other embodiments, 

effected media objects caused by an object included in the selection might be added 
to the tree instead or in addition to such causal media objects; 

iv) one constraint which might be added to each selection or combination of 
10 selections could be a constraint on the number of times a given media element 

appears in the finished media article. It might, for example, be dictated that any 
media element should only appear once in a given media article. 
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1 . A method of composing a media article comprising: 

5 analysing digital metadata associated with a first media file, which digital 

metadata includes: 

related file identity data identifying a second media file; and 
relationship data which indicates the relationship between what is 
represented by the first media file and what is represented by the second media file; 
10 and 

arranging said first and second media files in a media, article in accordance 
with said analysis. 

15 2. A method according to claim 1 further comprising generating said file identity 
data and said relationship data. 

3. A method according to claim 1 wherein said metadata further comprises data 
characterising what is represented by said media files, said method further 

20 comprising the step of selecting a subset of media files including said first and 
second media files from a set of media files. 

4. A method according to claim 3 comprising: 
25 making a plurality of such selections; and 

concatenating the results of said selections. 

5. A method according to any preceding claim wherein said arranging step 
30 arranges said media files so as to determine whether the user sees or hears what is 

represented by the first media file before or after he sees or hears what is 
represented by the second media file. 
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6. A method according to any preceding claim in which said media file contains 
video data. 

7. A media article composition apparatus comprising: 

5 

one or more memory devices storing, for each of a plurality of media files, metadata 
including relationship data indicating one or more relationships between the content 
represented in said media file and the content represented in one or more other media 
. files; and 

10 

one or more processors in communication with said one or more memory devices and 
arranged in operation to compose a media article by arranging said media files or 
identifiers thereof in accordance with said relationship data. 

15 8. An apparatus according to claim 7 in which said relationship data indicates a 
causal relationship between what is represented by one of said media files and what 
is represented by another of said media files. 

9. An apparatus according to claim 7 wherein said one or more processors is 
20 further arranged in operation to provide a user with an interface enabling the user to 

enter said relationship data. 

10. An apparatus according to claim 9 wherein: 
said metadata is stored in a database; and 

25 said one or more processors are further arranged in operation to query said 

database to obtain identifiers of media files whose metadata meets one or more 
conditions specified in said query. 

11. An apparatus according to claim 10 in which said database comprises an 
30 object-oriented database and metadata for each media file is stored as an object in 

said object-oriented database. 
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12. An apparatus according to claim 1 1 in which said relationship data is stored 
as data which defines the relationships between objects in the database. 

13. An apparatus according to claim 12 in which membership of a set -is 
5 indicated by each member of the set inheriting from a container object. 

14. An apparatus according to claim 7 further comprising a content store storing 
a plurality of media files, said metadata for each media file including a pointer to the 
location of said media file in said content store. 

10 

15. An apparatus according to claim 7 wherein: 

said one or more memories further store one or more media element selection criteria; 
and 

15 

said one or more processors are further arranged in operation to receive a set of 
media element identifiers and select said input set by selecting a subset of media 
element identifiers in accordance with said selection criteria; 

20 1 6. . An apparatus according to claim 1 5 wherein: 

said one or more media element selection criteria comprise a set of template data, 
each of said sets of template data listing a plurality of slots to be filled, and, for each 
slot, one or more associated requirements of media elements for filling said slot; and 

25 

said one or more processors are further arranged in operation to provide said subset 
by, for each of said slots, retrieving one or more identifiers of media elements whose 
metadata accords with said one or more requirements for said slots. 
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ABSTRACT 
MEDIA ARTICLE COMPOSITION 



A method and apparatus for the automatic composition of media articles is disclosed. 
5 Previous attempts to do this have involved the manual production of a detailed 
framework listing the types of media elements that might be inserted into the slots 
specified in that framework and the subsequent selection of media elements for those 
slots. The present invention improves upon this by associating metadata (51) with 
the media elements in a content store, which metadata includes relationship data 
10 indicating how what is portrayed by that media element relates to what is portrayed 
by one or more other media elements. Because relationship metadata is included 
media articles can be generated automatically with the need for the production of a 
detailed framework describing that media article. 
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